Skip to content

Meter, ledger, and gate: enforce AgentBudget at the LLM seams#554

Merged
xmap merged 6 commits into
mainfrom
worktree-budget-metering
Jul 12, 2026
Merged

Meter, ledger, and gate: enforce AgentBudget at the LLM seams#554
xmap merged 6 commits into
mainfrom
worktree-budget-metering

Conversation

@xmap

@xmap xmap commented Jul 11, 2026

Copy link
Copy Markdown
Owner

Summary

Lifts the 2026-06-20 AgentBudget enforcement pause (user-authorized 2026-07-11) by building the three missing pieces of the tier-1 coarse post-hoc ladder, then applying a five-reviewer gate panel's fixes.

Meter (6821181): the PRICING table now covers the whole fleet (opus-4-8/4-7, sonnet, haiku) with cache-write corrected to the 1h-TTL 2x tier CORA's producers actually pin, and telemetry emits gen_ai.provider.name per current OTel GenAI semconv (gen_ai.system is deprecated). A fitness test derives coverage from the fleet's default-model constants so an unpriced default can never silently meter $0 again.

Ledger (00e7c69): the per-call cost_usd the adapter already computed (and discarded after the histogram) now persists on entries_decision_inferences, guarded by a CHECK rejecting negative/NaN/infinite cost. Additive forward-only migration; legacy rows stay NULL and sum as $0.

Lookup (82254ef): SpendLookup port (consumer-shaped Protocol + frozen result VO + always-zero stub) with a Postgres adapter owned by the Decision BC, summing usd/tokens/calls per agent per half-open window. Overflow-proof SUM (numeric cast + bigint clamp).

Gate (7ddaa94): find_budget_breach checks monthly_usd_cap (UTC calendar month) then daily_token_cap (UTC calendar day) at the RunDebriefer and CautionDrafter seams, windowed on the triggering event's occurred_at so replays gate identically. A breach writes a deferred Decision (AgentBudgetExhausted) instead of calling the LLM; a lookup error propagates (never fails open). Both subscribers also gained the Versioned-only lifecycle gate the AgentStatus contract always documented, making suspend_agent actually stop the agent; the operator-triggered regenerate refuses for a suspended agent (AgentSuspendedError) but stays deliberately budget-ungated (accountable human, still metered and debited).

Review fixes (2bbbf60): the panel's MAJORs, plus the integration seeds now mirror the production bootstrap-then-promote ceremony (promote_seeded_agent) since Defined agents no longer fire.

Known limitations (documented in the port docstring, follow-ups filed)

  • Producer-supplied agent_id on inference rows is not yet bound to the calling principal Fixed in cd097a8: the handler now rejects any entry whose agent_id differs from the calling principal (InferenceAgentMismatchError, 403), whole batch refused before any write; the delegating recorder logs a mismatch loudly. Residual precondition: legacy-mode X-Principal-Id trust means the binding is as strong as principal authentication (bearer-auth mode ignores the header).
  • The Operation BC steering brain remains ungated and unledgered; per-call pre-estimate tier + durable steering-spend record are the next slice.
  • Coarse tier undercounts by design (NULL-cost legacy rows, unpersisted cache tokens): permissive, never spuriously blocking.

Test plan

  • unit: 12228 passed (gate windows/breach logic, both subscribers' budget + lifecycle gates, factory wiring pins, pricing table incl. fleet-coverage derivation)
  • integration: 1044 passed (Postgres spend sums incl. half-open boundary + agentless rows, subscriber end-to-end with promote ceremony, cost-overrun scenario)
  • contract: 3334 passed; architecture: full pass (OpenAPI snapshot regenerated)
  • naming-r3-reviewer: 18 new names, all PASS

🤖 Generated with Claude Code

xmap and others added 5 commits July 11, 2026 01:35
The cost meter was lying for two of the three models the fleet can
actually spend on: claude-opus-4-8 had no PRICING entry and the
steering brain's default claude-sonnet-4-5 had none either, so both
metered $0 with only a once-per-process warning; claude-opus-4-7
carried the pre-drop $15/$75 prices (Opus is $5/$25 since the 4.7
generation). Correct metering is a precondition for the budget
enforcement seam that follows: a gate fed $0 costs never fires.

The span and metric emission still used gen_ai.system, which the OTel
GenAI semconv deprecated in favour of gen_ai.provider.name; the durable
Decision-side inference record was already on the new name, so CORA was
split across the two. The kwarg follows the attribute (system ->
provider_name).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The per-call USD cost was computed at the adapter, emitted to the
cora.agent.llm.cost.usd histogram, and then discarded: the meter
existed, the ledger sink did not. Budget enforcement (unpaused
2026-07-11) needs a durable per-(agent, window) spend fact to sum, and
the inference entry is its natural home, since it already carries
agent_id, occurred_at, and the token counts the cost derives from.

Each LLM producer (RunDebriefer, CautionDrafter, regenerate slice) now
recomputes the cost via compute_cost_usd, the same pure function behind
the histogram, and carries it on AgentInferenceTrace through the
append_inferences slice onto a new nullable cost_usd column. Legacy
rows stay NULL ("not recorded then", distinct from a true $0 call).
Additive everywhere: command, REST body, MCP tool, migration, and the
regenerated OpenAPI snapshot.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The budget gate needs one question answered before it permits an LLM
call: how much has this agent already spent in the cap's window. This
adds the cross-BC port that answers it, in the exact shape of the
start_run lookup family: consumer-shaped Protocol + frozen result VO +
always-pass stub in cora.infrastructure.ports, production adapter
shipped by the BC that owns the data. Decision BC owns the durable
spend fact (entries_decision_inferences carries agent_id, occurred_at,
token counts, and now cost_usd), so PostgresSpendLookup lives in its
new adapters package and SUMs directly; a proj_agent_spend read model
stays deferred-with-trigger until the gate's read shows up in p95.

The port is window-agnostic (the consumer computes half-open bounds
per cap) so a future award-window allocation is a consumer-side change
only. NULL costs and missing token counts sum as zero, keeping the
coarse tier permissive rather than spuriously blocking. Kernel default
is AlwaysZeroSpendLookup, so declared caps cannot block tests that
don't exercise gating.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
The declared AgentBudget caps were recorded-only: nothing read them at
invocation, and, worse, a Suspended agent kept making LLM calls, since
the subscribers checked only Access-level Actor deactivation. The
operator-pause story (suspend on cost overrun, tighten, resume) was a
recorded intention, not a control. Enforcement was unpaused by explicit
user decision on 2026-07-11; this is the coarse post-hoc tier of the
enforcement ladder for the two serialized subscriber agents.

Two per-apply gates in RunDebriefer and CautionDrafter, mirrored:

- Suspension gate (pre-lease): a Suspended agent takes no actions, no
  LLM call, no Decision, until resumed. One Agent fold serves both
  gates.
- Budget gate (post-lease, so only the lease winner pays the lookup):
  cora.agent._budget_gate sums recorded spend via SpendLookup over the
  cap's UTC calendar window (month for monthly_usd_cap, day for
  daily_token_cap), computed from the terminal event's occurred_at so
  replays gate identically. A breach skips the call and records the
  refusal as the Run's deferred Decision (DebriefDeferred / NoAction,
  failure_error_class AgentBudgetExhausted), preserving the
  one-Decision-per-terminal-Run invariant so operators see WHY.

Overspend is bounded to about one in-flight call per caller, the
documented coarse-tier bound. A zero cap now refuses every call,
making the AgentBudget VO's recorded no-spend intent real. The
per-call pre-estimate tier for the steering brain follows separately
(it needs a neutral port; Operation BC cannot import the Agent BC
gate).

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Five-reviewer panel (3 baseline + security + test) on the four metering
slices surfaced MAJORs that all trace to one theme: the gate must be as
trustworthy as the caps it enforces.

- Cache-write pricing corrected to 2x base: producers pin 1h-TTL cache
  breakpoints and Anthropic bills 1h writes at 2x, so the old 1.25x
  tier under-metered exactly the calls CORA makes most.
- Lifecycle gate is Versioned-only (was Suspended-only): Defined means
  "not yet ready for invocation" and Deprecated is terminal per the
  AgentStatus contract, so both now skip, not just Suspended. The
  integration seeds gained an explicit bootstrap-then-promote step
  (promote_seeded_agent) mirroring the production ceremony the
  cost-overrun scenario already documented.
- Operator regenerate now refuses for a SUSPENDED agent
  (AgentSuspendedError): suspend means stop on every path, not just the
  subscriber path. Budget stays deliberately ungated there (an
  accountable human asked; the call is still metered and debited), now
  pinned by test.
- Spend SUM overflow-proofed (numeric cast + bigint clamp) and the
  append_inferences boundary bounded (token le=1e9, cost finite
  nonnegative) so one hostile row cannot poison a window.
- Migration CHECK rejects negative/NaN/infinite cost_usd at the ledger.
- Docstrings now carry the trust boundary (producer-supplied agent_id
  unbound to principal, follow-up owed), the fail-closed lookup-error
  posture, and the steering brain's ungated-and-unledgered deferral.

Test retrofit: versioned seeds throughout, factory wiring pins for
spend_lookup on both subscribers, half-open window boundary rows and
agentless-row exclusion on the Postgres lookup, deprecated-skip and
under-cap-proceeds parity on both subscribers.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Jul 11, 2026

Copy link
Copy Markdown

Coverage report

Click to see where and how coverage changed

FileStatementsMissingCoverageCoverage
(new stmts)
Lines missing
  apps/api/src/cora/agent
  _budget_gate.py
  apps/api/src/cora/agent/adapters
  anthropic_llm.py
  apps/api/src/cora/agent/aggregates/agent
  state.py
  apps/api/src/cora/agent/features/regenerate_run_debrief
  handler.py
  apps/api/src/cora/agent/subscribers
  caution_drafter.py
  run_debriefer.py
  apps/api/src/cora/api
  _inference_recorder.py
  main.py
  apps/api/src/cora/decision
  errors.py
  routes.py
  apps/api/src/cora/decision/adapters
  __init__.py
  postgres_spend_lookup.py
  apps/api/src/cora/decision/features/append_inferences
  command.py
  handler.py
  route.py
  apps/api/src/cora/infrastructure
  deps.py
  apps/api/src/cora/infrastructure/observability
  gen_ai.py
  apps/api/src/cora/infrastructure/ports
  __init__.py
  inference_recorder.py
  spend_lookup.py
Project Total  

This report was generated by python-coverage-comment-action

Closes the budget-ledger inflation hole the gate review flagged: the
append_inferences slice accepted a producer-supplied agent_id that was
never checked against the caller, and the AgentBudget gate sums spend
rows by agent_id, so any AppendInferences-authorized producer could
write rows against a victim agent's id and starve its budget (denial
of service by ledger inflation).

The handler now rejects any entry whose agent_id differs from the
calling principal (InferenceAgentMismatchError, HTTP 403), whole batch
refused before any write: no logbook open, no partial append, and no
decision-existence oracle since the check precedes the load. Rows with
agent_id=None (operator/tool provenance, no agent claim) are untouched,
and all three internal recorders already self-report with the agent's
own actor id as principal, so nothing legitimate changes shape.

Binding beats the persist-principal-and-filter alternative because a
sum-self-reported-only ledger would let an agent evade its own cap by
routing calls through a helper producer; rejecting mis-attributed rows
outright closes inflation without opening that evasion path. A future
on-behalf-of recorder (Tier 1.5 steering spend) gets an explicit
delegation mechanism, not a relaxation.

The DelegatingInferenceRecorder logs the mismatch loudly and distinctly
(a mis-wired internal agent would otherwise silently drop spend rows,
eroding the gate fail-open). Security review verified no bypassing
write path, no UUID case/whitespace normalization asymmetry between the
handler equality and the gate's SQL, and no info leak in the 403;
residual precondition: legacy-mode X-Principal-Id trust means the
binding is as strong as principal authentication, per the documented
trust-the-proxy posture.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
@xmap xmap merged commit 2f825f7 into main Jul 12, 2026
16 checks passed
@xmap xmap deleted the worktree-budget-metering branch July 12, 2026 02:22
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant